Understanding Technical Articles and Their Diagrams

– A comprehensive system is being developed to transform scientific research papers into a knowledge base. An intelligent Scientist's Assistant is also being developed to allow a scientist to navigate through the knowledge base. Starting from paper documents, both text and graphics are captured and analyzed. We describe the entire system and then focus on the analysis of diagrams such as data graphs. Model-based image processing discovers graphical objects such as lines, polygons and text. The model-based methods are necessary to deal with occlusion such as data points lying on data lines, or data lines which collide with graph axes. The analysis is assisted by the use of a spatial index. Then a syntactic Graphics Constraint Grammar (GCG) is used to parse the collection of objects to identify higher-level structures. Each grammar rule has three components: (1) a production specifying a parent and its constituents, (2) a set of constraints which are typically geometrical relations that the constituents must satisfy, and (3) a set of propagation rules which specify the mapping between properties of the parents and of the children. The complexity of the parsing and its constraint satisfaction problem is reduced by an early grouping of similar elements (such as data point symbols or tick marks on scale lines) into approximate equivalence classes by using Generalized Equivalence Relations (GERs). Computation of the GERs is very efficient due to the use of a data structure (GOSSAMER) similar to the one used at the image processing stage. A semantic GCG grammar is then used, starting with the syntactic objects already found, to generate a frame-based knowledge representation of the contents of the diagrams. The article ends by discussing the problems that will be solved and those that will remain when all documents become electronic. A comprehensive document understanding system is described which analyzes diagrams by parsing with Graphics Constraint Grammars. Combining this with natural language analysis results in a knowledge base which represents the meaning of the document. The design of a comprehensive document image analysis system should be driven by the ultimate use of the information in the document. If huge quantities of text are being processed for later retrieval, then handling special notation such as 2.4x10 7 or H 2 0 may not be important. Similarly, if each figure in a document is going to be accessed as a single entity, then only a raster version needs to be …